Outgoing Spam Monitoring

The SpamExperts filters are extremely effective at blocking a large percentage of outgoing spam/viruses, to prevent issues with your network reputation. However, it is essential that you are proactive in stopping the abuse at its source by suspending any spamming customers/accounts. If such accounts are not suspended/blocked, there may eventually be a spam run which is missed by our engines. You can prevent any such spam escalations (or other type of attacks from abusive customer accounts), by ensuring the account is locked down before it starts to cause real issues. Our systems allow you to quickly and easily identify such abusive accounts, before any third-party issues occur.

There are a number of ways that spammers can be monitored via our systems.

Control Panel Quarantine (Local Cloud Only)

If you have super administrator access to the control panel, you can review the blocked outgoing spam emails on the "Spam quarantine" page in the Outgoing section as Super Administrator. Although the number of daily spam emails you find there can be overwhelming at start, spending 15 minutes a day to analyze/block the source of the most frequent messages you can find there will quickly result in a significant drop of overall spam traffic.

This is strongly recommended when starting to use our filtering, so your administrators can easily pinpoint the top spam causes and get more familiar with tracking down/blocking the spam sources.

Best Practice for Smarthost Users

Ensure all your smarthost authentication users are grouped as part of a single administrative domain (e.g. out.yourcompany.tld)
Configure your sending MTA to always include an end-user identification header
Set your outgoing user account to use this identity header
Manually/automatically locate abusive identities and shutdown the main spam source (and temporarily lock down the identity via our identity management as an immediate measure)

Managing Outgoing Spam

Outgoing Log Search

You can view outgoing blocked messages from the Admin, Domain or Email Level Control Panel using the Outgoing Log Search:

Select Outgoing > Logs
In the Query Rules panel, filter using Status > is one of > Quarantined

Click Show Results to list all matches

Outgoing Reports page

You can view senders/Identities in grouped format using the Outgoing Reports feature from the Admin or Domain Level or Control Panel.

Select Reporting > Outgoing Reports
Select the relevant domain if accessing from the Admin Level
Select the Period
In Classification, select Rejected (or Accepted if you wish to see accepted emails and not quarantined ones)
In the Group by dropdown, select identity
Click Show to display all results

Classifications

A list of Message Classifications can be found by clicking on Classifications in the description at the top of the Log Search page.

Classifications relate to the message Sub Classes that can be found by adding the Sub Class and Extra Class columns to the view using the Columns to be displayed dropdown.

More information on what classifications mean and how these can be dealt with can be found here:

What do SMTP Message Responses Mean?

Automatic and Manual Locking

Manually Lock Identity from the Outgoing Reports Page

Click the lock icon next to the identity
To unlock the identity, click the lock icon again

Automatic Locking based on the Identity header within the Control Panel

You can choose to auto-lock senders based on their Identity header. For this to work, there must first be a configured Identity.

To start autolocking senders based on this you need to make sure the option Lock Identities Automatically:" is set to "Yes" in the outgoing user settings page:

Select Outgoing > Manage users
Locate the outgoing user you want to configure, and from the dropdown, select Edit. The Outgoing user settings page is displayed
Ensure the Lock identities automatically option is set to Yes:

Identities will be locked when a certain amount of spam, phishing or virsues is seen in a short time frame.

The locked identities can continue to be seen via the log search and outgoing reports page.

Manually Lock Identity from Outgoing Log Search

You can choose to lock a sender based on their identity header from this page:

Locate the relevant message and select Lock Identity from the dropdown:

In the prompt, enter a reason for locking this sender and click Confirm

You can also lock the Outgoing user from here (by selecting Lock user in the dropdown). This would prevent any outgoing mail being sent from that outgoing user (IP or domain).

Manual locking from the Manage Identities page

The manage Identities page gives you a comprehensive overview of the current locked and unlocked identities. To lock or unlock either on mass or individual identities you can do the following:

Use the drop down menu next to the Identity
Click lock
Enter a reason for the lock
Click Execute

To unlock, do the reverse.

Manual Locking senders based on the identity header from the API

When using IP authentication, it's often needed to be able to lock specific senders without locking the whole IP. This can be done by locking senders via their Identification method (Envelope-sender, Authenticating User, Identification Header). To do this you need to execute the following API:

https://apihostname/cgi-bin/api?call=api_lock_outgoing_identity&domain=DOMAIN&identity=bob@example.com&username=USERNAME

To list the current locked users:

https://apihostname/cgi-bin/api?call=api_list_locked_identities&domain=DOMAIN&username=USERNAME

Please note, to be able to use this method, an Identification header must first be set as mentioned above.

Alternative reporting

Outgoing Spam Reports via CSV

There is an option to have a daily CSV report for outgoing spam per outgoing user. The API call to activate the daily report with spamming accounts is: api_set_outgoing_report_recipient(domain, recipient='', username='') > "".

Set the address where the outgoing filter report should be sent to. If the 'recipient' argument is omitted then disable this feature. Please note that this feature is in development and the format and content of the report are subject to change without our usual deprecation procedures.

https://MASTERHOSTNAME/cgi-bin/api?call=api_set_outgoing_report_recipient&domain=DOMAIN&recipient=RECIPIENT&username=USERNAME"

So this should be good for you in regards to the monitoring of outgoing spam, and be able to see overall information. (please note , subject and body will not be shown here)

Alternative Outgoing Spam Reports via CSV based on "Identification header"

There is another option to have a CSV report for outgoing spam per identification header sender that is sent every 2 hours to a specified email address. The API call to activate the daily report with spamming accounts is: api_set_outgoing_report2_recipient(domain, recipient='', username='') > "".

https://master.hostname/cgi-bin/api?call=api_set_outgoing_report2_recipient&domain=DOMAIN&username=USERNAME&recipient=RECIPEINT

The report will contain counts of blocked spam per Identification and counts of invalid senders.

For example:

"Authentication Domain","Authentication User","User Identification","Spam Count","Invalid Sender Count"
example.com,,bob@example.com,100,0
example.net,,example.net,235,301
example.org,,example.org,0,2000

IMAP quarantine access

Rather than using any of the scripts, or the Spampanel webinterface, you can authenticate with your "global" administrator account (Local Cloud only) using any IMAP compatible email client for real-time access to the spam quarantine. Please contact our support in case you do not have the "global" credentials yet.

Global quarantine reporting script

Rather than using Spampanel or direct IMAP quarantine access to review the quarantine, this is a script that will parse the outgoing IMAP quarantine.

(Local Cloud Only) You can download this here: http://download.seinternal.com/tools/retrieve_quarantine_info.py

Usage: retrieve_quarantine_info.py [options]

Output a list of quarantined outgoing messages.

Options:
-h, --help show this help message and exit
-c, --csv saves output to csv file
-d, --display displays the loglines as they pass by
-i, --incoming search the incoming quarantine
-o, --outgoing search the outgoing quarantine
-t, --today load results from today (otherwise yesterday)
-s IMAPHOST, --imaphost=IMAPHOST
The hostname of the imap server
-u IMAPUSER, --username=IMAPUSER
The username to check, usually 'global'
-p IMAPPASS, --password=IMAPPASS
The password for the 'global' user
-n, --no-bounce filters out mail originating from 'mailer-daemon'

Please make sure you run this from a NON FILTERING server only. This will retrieve the quarantined messages either in an on screen format or saved to a .csv file. For example you could do something like this: $./retrieve_quarantine_info.py -d -o -n -s MASTERHOSTNAME

This will display on screen the messages that have been quarantined outgoing in the last hour. You will be prompted for a "Global" password. This is given out only on request. If this is required please contact our support team by raising a support ticket through N-AbleMe.

This will show you something like this:

$./retrieve_quarantine_info.py -d -o -n -s MASTERHOSTNAME (or quarantine server if applicable)
Please enter the password for the IMAP account 'global'.
Password:
#,From,To,Reply-To,Qmail UID,Invoked for,IP/Username,Evidence,PHP script,Auth.sender,Auth-User,Auth-Email
1,--DATA WILL BE HERE--
2,--DATA WILL BE HERE--
3,--DATA WILL BE HERE--
4,--DATA WILL BE HERE--
5,--DATA WILL BE HERE--

Here you can see the information on the blocked messages and some relevant details. Alternatively it may be easier for you not to display the data and save it to a csv file, then you can open it in any excel like program and sort it on the specific field to group the data. This should then give you a better idea of some of the clients that you can close down for spamming.

Using the Log Search API

It's possible to use the api_find_outgoing_messages to be able to get for example a list of the top 50 spammers in X amount of time. A bash example can be seen below:

curl -k -s "https://user:pass@master.hostname/cgi-bin/api?call=api_find_outgoing_messages&domain=DOMAIN&from_date=`date -d '12 hours ago' +'%s'`&to_date=`date +%s`&predicate=and&partial=False&sort_field=datestamp&classification=oversize%2Cblacklisted%2Clocked%2Cphish%2Cvirus%2Cspam%2Cdeferred&include_in_progress=False&id=&subject_header=&api_language=en&columns=sender" | sort | uniq -c | sort -nr | head -50

It's also then possible to start automating actions. For example, you can use 2 API's 'api_find_outgoing_messages' and 'api_blacklist_outgoing_sender' to be able to query the log API to find senders that have sent X amount of messages in X amount of time, and then take a further action. The example below shows a way to check the outgoing logs for the last 24 hours, and if the sender has sent more than 2000 messages then block the sender.

curl -s -k "https://user:pass@master.hostname/cgi-bin/api?call=api_find_outgoing_messages&domain=DOMAIN&from_date=`date -d '12 hours ago' +'%s'`&to_date=`date +%s`&predicate=and&partial=False&sort_field=datestamp&include_in_progress=False&columns=sender" | sort | uniq -c | grep "@" | sort | awk '{if($1==$1+0 && $1>2000)print $2}' | xargs -I{} curl -k -s "https://user:pass@master.hostname/cgi-bin/api?call=api_blacklist_outgoing_sender&domain=DOMAIN&sender={}'

While these are very basic examples, our API is very versatile, so these can be amended or changed to suit your exact needs, for example, by locking specific senders based on the identification headers or more.

ARF reports

An ARF report is sent each time an outgoing spam message is blocked, and will contain a copy of the original message including headers. For information on how to set this up, see Configure the Abuse Report Address.

Many larger companies already process ARF reports originating from external sources such as AOL. You can set your administrator address to point to your existing ARF parsing infrastructure, so your existing abuse handling systems automatically receive and process our datafeeds.

ARF parser

If you do not have an ARF parser yet, we recommend that you set up a system to handle your incoming ARF reports. We can recommend the free opensource software Abuse.IO for this. Alternatively you can e.g. use a python file that can parse the contents of the ARF reports. Your sysadmins will know how best they can utilize this and parse the data that they need.

Using ARF automation also allows you to accept ARF feed from third-parties, to further improve your abuse handling and to deal with abuse that does not (yet) use our outgoing filter.